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[Document Name] Specification 

[Title of the Invention] Image Retrieval Information Storing Apparatus and Image 
Retrieving Apparatus 
[Scope of Claims for Patent] 
[Claim 1] 

An image retrieved information storing apparatus, comprising: 

a coding information reading unit reading prescribed coding information which 
will be information representing frame feature from coded image data; 

a frame feature value generating unit obtaining a frame feature value which is a 
numerical representation of the frame feature, based on said coding information; and 

a storing unit storing said coding information and frame feature value in 
correspondence with each frame of said coded image data. 

[Claim 2] 

An image retrieval information storing apparatus, comprising: 

a coding information reading unit reading prescribed coding information which 
will be information representing frame feature from coded image data; and 

a storing unit storing said coding information in correspondence with each frame 
of said coded image data. 

[Claim 3] 

An image retrieval information storing apparatus, comprising: 

a coding information reading unit reading prescribed coding information which 
will be information representing frame feature from coded image data, 

a frame feature value generating unit obtaining a fr^e feature value which is a 
numerical representation of the frame feature, based on said coding information; and 

a storing unit storing said frame feature value in correspondence with each frame 
of said coded image data. 

[Claim 4] 

An image retrieving apparatus, comprising: 
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an input unit receiving as input, coded image data, prescribed coding information 
which will be information representing frame feature stored in correspondence with each 
frame of the image data, and a frame feature value which is a numerical representation 
of the frame feature based on the coding information; 

a frame feature value generating unit obtaining a frame feature value which is a 
numerical representation of the frame feature, based on said coding information; 

an index information generating unit determining a featured frame among said 
image data based on the frame feature value from said input unit or frame feature value 
generating unit and generating index information which is positional information of said 
featured frame; and 

an output unit outputting an index image based on the index information. 

[Claim 5] 

An image retrieving apparatus, comprising: 

an input unit receiving as input, coded image data and prescribed coding 
information which will be information representing frame feature stored in 
correspondence with each frame of the image data; 

a frame feature value generating unit obtaining a frame feature value which is a 
numerical representation of the frame feature, based on said coding information; 

an index information generating unit determining a featured frame among said 
image data based on the frame feature value from said frame feature value generating 
unit and generating index information which is positional information of said featured 
frame; and 

an output unit outputting an index image based on the index information. 
[Claim 6] 

An image retrieving apparatus, comprising: 

an input unit receiving as input, coded image data and a frame feature value 
which is a numerical representation of frame feature stored in correspondence with each 
frame of the image data; 
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an index information generating unit determining a featured frame among said 
image data based on the frame feature value and generating index information which is 
positional information of said featured frame; and 

an output unit outputting an index image based on the index information, 
[Detailed Description of the Invention] 

[Technical Field to Which the Invention Belongs] 

The present invention relates to generation and storage of image retrieval 
information used for retrieving a desired image from accumulated plurality of pieces of 
coded image data and to a method of implementing image retrieval using the image 
retrieval information. 

[Prior Art] 

When a motion picture is retrieved, it is important to recognize the contents of 
the motion picture, that is, what is the object picked up in the motion picture. To that 
end, except when a text title or description is provided, in most cases it is necessary for 
the user to confirm the contents by reproducing the whole motion picture. As the 
number of motion pictures among which a desired one is to be retrieved increases, it 
becomes harder for a user to reproduce the motion pictures one by one and to confirm 
the contents thereof, and the efficiency of retrieval is quite poor, EfiRciency further 
lowers when each motion picture is in the form of coded compressed data requiring 
decoding and reproduction before retrieval. In order to solve this problem, a method 
of retrieval may be effective in which one or a plurality of frames characteristic of the 
motion picture, typically representing the contents of the motion picture are extracted 
and displayed, so that the contents of the motion picture can be displayed in schematic 
manner. 

The characteristic frame (hereinafter referred to as featured frame) used for 
motion picture retrieval includes, for example, a scene change frame. Japanese Patent 
Laying-Open No. 9-261648 describes a method in which a scene change frame is 
extracted, based on prediction mode information included in image data, from image 
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data coded in accordance with interframe prediction coding. Over one frame, the 
number of I blocks which are coded in accordance with intraframe prediction, P blocks 
coded in accordance with forward interframe prediction, B blocks coded in accordance 
with backward interframe prediction and Bi blocks coded in accordance with 
bidirectional interframe prediction are counted. Here, as frequency information of 
these respective blocks reflects correlation among frames in the motion picture, based 
on the frequency information of each block, a frame immediately following a scene 
change (== scene change frame) is detected. Further, Japanese Patent Lajdng-Open No. 
10-23421 describes a method of extracting a scene change frame, based on the change in 
code vfdue of each piece of coding information contained in the coded image data, A 
conventional image retrieval information storing apparatus in accordance with the prior 
art is as shown in Fig. 14. 

A coding information reading unit 1401 reads, according to Japanese Patent 
Laying-Open No. 9-261648^ block by block prediction mode information from coded 
image data, and reads, according to Japanese Patent Laying-Open No. 10-23421, 
respective pieces of coding information of different types. Then, a frame feature value 
generating unit 1402 calculates, from the various pieces of coding information read by 
coding information reading unit 1401, frequency of respective prediction mode blocks 
according to Japanese Patent Laying-Open No. 9-261648, and calculates accumulated 
value of code value of respective pieces of coding information according to Japanese 
Patent Laying-Open No. 10-23421. An index information generating unit 1303 
determines the scene change frame which is the featured frame, based on the frame by 
frame feature value obtained in a frame feature value generating unit 1302, that is, 
frequency information of respective prediction mode blocks according to Japanese 
Patent Laying-Open No- 9-261648, and accumulated code value of respective pieces of 
coding information according to Japanese Patent Laying-Open No. 10-23421 and 
outputs positional information thereof as index information. Finally, an index 
information storing unit 1404 stores the index information output from index 
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information generating unit 1403 as image retrieval information. On the other hand. 
Fig. 15 represents a conventional image retrieving apparatus using the image retrieval 
information stored in this manner. An image retrieval executing unit 1501 executes 
actual retrieval using the image retrieval information and the coded image data as inputs 
and outputs the results of retrieval. 

The featured frame includes, in addition to the scene change frame, a scene 
change frame based on rapid/slow movement, described in Japanese Patent Laying-Open 
No. 9-284702. According to Japanese Patent Laying-Open No. 9-284702, a motion 
vector for detecting a scene change frame is independently obtained from the original 
image, not from the coding information, and the scene change frame is detected based 
on the total sum of motion vectors in a frame. 

[Problems to be Solved by the Invention] 

In the above-described conventional image retrieval information storing 
apparatus, positional information of the extracted featured frame is obtained, and the 
positional information only is stored as the index information. Therefore, even when a 
plurality of images which are the objects of retrieval have mutually different features, 
only a featured frame representing one same feature only can be obtained. If feature 
frames reflecting features of respective images are to be obtained, it is necessary to go 
back to the coded image data and generate the image retrieval information. In the 
process of retrieval, when featured frames corresponding to various features of images 
which are the object of retrieval are to be taken out in an interactive manner, going back 
to the coded image data and generating the image retrieval information significantly 
lowers retrieval efficiency. Further, preparation of featured frame indexes 
corresponding to every possible feature for all the image data is not practical. Further, 
it is not possible by the prior art to meet with high efficiency a request for a featured 
frame which has not been expected by the provider providing the image retrieval 
information. 

The present invention was made in view of these problems, and the present 
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invention allows adaptive extraction of a featured frame in a process of retrieval by 
storing coding information utilized for extracting a featured frame or a feature value for 
each frame for extracting a featured frame. 
[Means for Solving the Problems] 

According to claim 1 of the present invention, the problem above is solved by 
including a coding information reading unit reading prescribed coding information which 
will be information representing frame feature from coded image data, a frame feature 
value generating unit obtaining a frame feature value which is a numerical representation 
of the frame feature, based on the coding information, and a storing unit storing the 
coding information and frame feature value in correspondence with each frame of the 
coded image data. 

According to claim 2 of the present mvention, the problem above is solved by 
including a coding information reading unit reading prescribed coding information which 
will be information representing frame feature from coded image data; and a storing unit 
storing the coding information in correspondence with each frame of the coded image 
data. 

According to claim 3 of the present invention, the problem above is solved by 
including a coding information reading unit reading prescribed coding information which 
will be information representing frame feature from coded image data, a frame feature 
value generating unit obtaining a frame feature value which is a numerical representation 
of the frame feature, based on the coding information, and a storing unit storing the 
frame feature value in correspondence with each frame of the coded image data. 

According to claim 4 of the present invention, the problem above is solved by 
including an input unit receiving as input, coded image data, prescribed coding 
information which will be information representing frame feature stored in 
correspondence with each frame of the image data, and a frame feature value which is a 
numerical representation of the frame feature based on the coding information, a frame 
feature value generating unit obtaining a frame feature value which is a nimaerical 
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representation of the frame feature, based on the coding information, an index 
information generating unit detennining a featured frame among the image data based 
on the frame feature value from the input unit or frame feature value generating unit and 
generating index information which is positional information of the featured frame, and 
an output unit outputting an index image based on the index information. 

According to claim 5 of the present invention, the problem above is solved by 
including an input unit receiving as input, coded image data and prescribed coding 
information which will be information representing frame feature stored in 
correspondence with each frame of the image data, a frame feature value generating unit 
obtaining a frame feature value which is a numerical representation of the frame feature, 
based on the coding information, an index information generating unit determining a 
featured frame among the miage data based on the frame feature value from the frame 
feature value generating unit and generating index information which is positional 
information of the featured frame, and an output unit outputting an index image based 
on the index information. 

According to claim 6 of the present invention, the problem above is solved by 
including an input unit receiving as input, coded image data and a frame feature value 
which is a numerical representation of frame feature stored in correspondence with each 
frame of the image data, an index information generating unit determining a featured 
frame among the image data based on the frame feature value and generating index 
information which is positional information of the featured frame, and an output unit 
outputting an index image based on the index information. 
[Embodiments] 

In embodiments of the present invention, a scene change frame and a frame in 
which rapid movement is locally observed are extracted as feature frames, by way of 
example, using motion vector information and coding prediction mode information 
among coding information included in the image data. A case where other coding 
information is used or a case where other feature frames are to be extracted may also 
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naturally have a configuration shown in the present invention. 

Figs. 1 to 7 show a first embodiment of an image retrieval information storing 
apparatus and an image retrieving apparatus of the present invention. 

Fig. 1 is a configuration diagram representing the first embodiment of the image 
retrieval information storing apparatus of the present invention. A coding information 
reading unit 101 receives coded image data as an input, and reads and outputs desired 
coding information included in the image data. Coding information reading unit 101 in 
the present embodiment extracts and outputs prediction mode information and motion 
vector information from the coded image data. 

A fi^ame feature value generating unit 102 processes each piece of coding 
information obtained fi-om coding information reading unit 101, and generates and 
outputs a feature value for each fi"ame as a reference for determination of a featured 
fi*ame. Fig. 2 illustrates an internal configuration of firame feature value generating unit 
102 in Fig. 1 in the present embodiment. 

Coding prediction mode information read by coding information reading unit 101 
is assumed as a prediction coding method for each fi-ame and a prediction method for 
each coding block in the fi-ame. A prediction mode counting unit 20 1 counts the 
number of blocks coded in accordance with respective prediction methods, fi^ame by 
fi-ame. In a forward prediction coding fi-ame (P fi^ame), the number of I blocks coded 
in accordance with intraframe prediction and the number of P blocks coded in 
accordance with interfi-ame prediction are counted, respectively. In a bidirectional 
prediction coding fi-ame (B fi"ame), the number of I blocks coded in accordance with 
intrafi-ame prediction, the number of P blocks coded in accordance with forward 
interfi-ame prediction, the number of B blocks coded in accordance with backward 
interframe prediction and the number of Hi blocks coded in accordance with 
bidirectional interfi"ame prediction are counted, respectively. 

Further, prediction mode counting unit 201 outputs, as block prediction mode 
fi^equency information for each frame, the number Pp of P blocks for the P fi-ame, and 
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the total number Bf (= P + Bi) of forward interframe prediction blocks and total number 
Bb (= B + Bi) of backward interframe prediction blocks for the B frame, 

A frequency information converting unit 202 converts these pieces of frequency 
information to attribute information for each frame (in the following, the attribute 
information related to each frame will be referred to as the "frame feature value"). In 
the present embodiment, each frame feature value is set such that it represents 
correlation between the frame of interest and an immediately preceding frame. 

Fig. 3 shows an example of the relation of reference among P and B frames at 
the time of interframe prediction coding with an arrow. A starting point of the arrow 
represents a frame which is an object of coding/decoding, and the end point of the arrow 
represents a reference frame at the time of coding/decoding. When the reference 
characters Pp, Bfh, and Bbn allotted to respective arrows represent the aforementioned 
frequency information, for example, the frame feature value for frame B2, that is, 
frequency information related to correlation between frames BI and B2 should be 
calculated by a prescribed operation from respective block frequency information Pp, 
Bfii, and Bbn allotted here. 

In the present embodiment, the frame feature value of frame B2 is the value of 
Bf2 when Bf2 exceeds a certain threshold value, the value of Bbl when BCZ does not 
exceed the certain threshold value but Bb 1 exceeds the threshold value, and otherwise 
the value of Bf2 or Bbl which is larger. The frame feature value of frame B3 is also 
calculated in the similar manner. As to frame B 1, the value of Bfl is used as the frame 
feature value. In addition, as to frame P4, the value of Bb3 or Pp which is larger is 
used as the frame feature value of frame P4. Naturally, the method of calculating 
attribute information for each frame is not limited to this. Further, the frequency 
information of the coded blocks counted by prediction mode counting unit 201 may be 
directly regarded as frame feature values. 

A motion vector statistic calculating unit 204 calculates an average value of the 
vector lengths of motion vectors corresponding to P block for the P frame over the 
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entire frame, based on the prediction mode information obtained by coding information 
reading unit 101 and an average value of vector lengths of motion vectors 
corresponding to all the blocks (Bf) coded in accordance with the forward interframe 
prediction and all the blocks (Bb) coded in accordance with the backward interframe 
prediction, for the B frame, over the entire frame. The statistics calculated may not be 
limited to the average value of vector lengths, and different statistics including average 
vector may be calculated. Further, the motion vector is normalized considering space 
between an object frame and a reference frame at the time of calculation. 

In addition, in the present embodiment, motion vector statistic calculating unit 
204 calculates the statistic by using only motion vectors determined as valid by a valid 
motion vector detecting unit 203. When a camera is moving while being focused on an 
object, for example, the object and the background of the picked up image may have 
motion vectors very much different in magnitude or direction. In such a case, valid 
motion vector detecting unit 203 is used to extract only that motion vector of an area of 
interest, for example, the area of the object, to utilize the extracted vector for 
determining magnitude or the like of motion. 

Possible method of separating the object area from the background may include 
a method in which an average of the motion vectors of the entire frame is calculated and 
the motion vectors out of the average are successively extracted, and a method in which 
areas are divided into two by clustering. If the contents of the motion picture come to 
be more complicated as the number of the objects increase, for example, a method of 
valid motion vector detection in accordance with the complexity may be used. If the 
image is not of such a nature, the statistic of the motion vector in the entire frame may 
be calculated, without valid motion vector detection. 

A statistic information converting unit 205 converts each of the above-described 
motion vector statistics obtained by motion vector statistic calculating unit 204 to a 
frame feature value provided for each frame. Similar to the frame feature value based 
on the prediction mode information, the frame feature value based on the motion vector 
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information in the present embodiment is set such that it indicates the motion 
information between a frame of interest and an immediately preceding frame At this 
time, the coding block frequency information obtained by prediction mode calculating 
unit 201 is referred to in calculating the motion vector frame feature value. 

In Fig. 3, regarding the motion vector frame feature value provided to frame B2, 
a vector length average of all motion vectors corresponding to B£2 and Bb 1 is provided 
when both of block frequencies Bf2 and Bbl exceed a certain threshold value, a vector 
length average of motion vectors corresponding to Bf2 is provided when only Bf2 
exceeds the threshold value, a vector length average of motion vectors corresponding to 
Bbl is provided when only Bbl exceeds the threshold value, and otherwise 0 is provided. 
This is also the case with frame B3 . As to frame B 1 , a vector length average of motion 
vectors corresponding to Bfl is provided when Bfl exceeds a certain threshold value 
and otherwise 0 is provided. In addition, as to frame P4, a vector length average of 
(normalized) all motion vectors corresponding to Bb3 and Pp is provided when both of 
Bb3 and Pp exceed a certain threshold value, a vector length average of motion vectors 
corresponding to Bb3 is provided when only Bb3 exceeds the threshold value, and 
otherwise 0 is provided. 

Naturally, the method of calculating the motion vector frame feature value for 
each frame is not limited to the above. The motion vector statistics corresponding to 
respective blocks described above calculated by motion vector statistic calculating unit 
204 may be regarded as motion vector frame statistics frame by frame. 

The frame feature value on prediction mode and the frame feature value on 
motion vector obtained in this manner are output from frame feature value generating 
unit 102 and stored in prescribed storing locations of frame feature value storing unit 
103. The storing location of the frame feature value may be a location corresponding 
to the coded image data of each frame, for example, and it may be stored as header 
information of the coded image data of each frame. Alternatively, the frame feature 
value may be stored as separate data for image retrieval, independent of the coded image 
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data. 

Fig. 4 is a configuration diagram representing a first embodiment of an image 
retrieving apparatus of the present invention. An index information generating unit 401 
receives as input the frame feature value stored in the image retrieval information storing 
apparatus, extracts a desired featured frame based on the frame feature value of each 
frame, and generates and outputs as the index information, the positional information of 
the featured frame. 

Fig. 5 illustrates an internal configuration of index information generating unit 
401 in Fig. 4 in the present embodiment. According to the input frame feature value on 
prediction mode related to coding, a scene change frame is determined by a scene 
change frame determining unit 501 . In the present embodiment, the scene change 
frame is determined by thresholding the frame feature value on prediction modes for 
coding. By changing the setting of the threshold value or by adding a process when an 
adjacent frame is determined to be the scene change frame, the scene change frame to be 
extracted varies. 

In the prior art, the process of extracting a scene change frame has been 
independent of the image retrieval process, and the method of determining and the like 
have been fixed. In the embodiment of the present invention, the threshold value for 
determining the scene change or the process of determination is adaptiveiy changed in 
the process of retrieval in response to a request for extracting a new^ featured frame 
output from an image retrieval executing unit 402 in the process of image retrieval, as 
v^U be described later. Accordingly, the desired scene change frames are extracted 
successively. A scene change frame index generating unit 502 generates, as index 
information, positional information of the scene change frame determined by 
determining unit 501, 

Using the frame feature value on motion vector as an input, a featured frame 
related to magnitude of motion or the like is determined by a motion featured frame 
determining unit 503. In the present embodiment, peak position determination is made 
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based on the frame featured values of motion vectors of neighboring plurality of frames, 
whereby a frame in which local abrupt movement is observed is determined. 
Alternatively, a frame having a magnitude not smaller than a threshold value may be 
extracted, or a frame of which motion is hardly observed and which is at a relative 
minimum position, may be extracted as the motion featured frame. 

In the prior art, the process of extracting a motion featured frame has been 
independent of the image retrieval process, and the method of determination and the like 
have been fixed. The embodiment of the present invention is configured such that the 
method of determining motion featured frame related to what motion feature is of 
interest is adaptively changed in the process of retrieval, in response to a request for 
extracting a new featured frame output from image retrieval executing unit 402 in the 
process of image retrieval, which will be described later, to successively extract desired 
motion featured frames. A motion featured frame index generating unit 504 generates, 
as index information, positional information of the motion featured frame determined by 
determining unit 503. 

The scene change frame index information and motion featured frame index 
information obtained in this manner are output from index information generating unit 
401 and input to image retrieval executing unit 402. 

Image retrieval executing unit 402 executes image retrieval based on the input 
featured frame index information and the coded image data. Positional information of 
the featured frame is obtained from the featured frame index information, and the 
corresponding frame is decoded using the coded image data. A person performing 
retrieval executes retrieval by recognizing the contents of the motion picture by 
thumbnail display (catalogue display) or respective featured frames displayed 
successively and searching for a desired image. At this time, when it is determined that 
the featured frame applied as index information is not suitable for image retrieval being 
executed or that the featured frame is insufficient, a request for extracting a new 
featured frame is issued from image retrieval executing unit 402 to index information 
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generating unit 401 . Receiving the request for extracting a new featured frame issued 
from image retrieval executing unit 402, index information generating unit 401 changes 
the method of determining featured frames in featured frame determining units 501 and 
503, determines a desired featured frame, and outputs the positional information of the 
featured frame as the index information. The output index information is input to 
image retrieval executing unit 502, and image retrieval is newly executed. 

The advantages of the present first embodiment are as follows. As described 
with respect to index information generating unit 401, the featured frame on motion 
pictures, such as the scene change frame and the featured frame on motion, may be 
extracted differently dependent on the manner of detennination, and therefore, physical 
significance of the frame may differ. For example^ a frame in which rapid movement is 
observed and a frame in which movement is hardly observed are frames of completely 
different features. 

When motion picture retrieval is executed, whether it is the best to display a 
frame in which rapid movement is observed, or a frame with less movement, or a frame 
having different features on motion as the featured frame of the motion picture depends 
on the motion picture which is the object of retrieval, and therefore, it is possible that 
there is a request for adaptive switching in the process of retrieval. In such a situation, 
the first embodiment of the present invention can immediately meet the request for 
switching the featured frame in the process of retrieval, as the image retrieval 
information is stored in the form of frame feature values. 

Alternatively, as shown in Fig. 6, it is also possible to store index information of 
(predetermined) featured frames together with the frame feature values by adding index 
information generating unit 401 generating index information of the featured frames and 
an index information storing unit 601 storing index information. In an example where 
the frame feature values and featured frame index information are both stored, a request 
for switching a featured frame in the process of retrieval can be immediately met by 
using the frame feature value, and high speed retrieval can also be executed by using 
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fixed index infonnation when there is no request for switching in the process of retrieval 

Fig. 7 shows a configuration of the image retrieving apparatus corresponding to 
the image retrieval information storing apparatus shown in Fig, 6. In the image 
retrieving apparatus in Fig. 7, the stored index information and the index information 
output firom index information generating unit 401 are both input to image retrieval 
executing unit 402 and used for image retrieval. 

In the present embodiment, the image retrieval information storing apparatus and 
the image retrieving apparatus are diflFerent apparatuses between which index 
infonnation and the firame feature values are stored and passed. Such information 
generated by the image retrieval information storing apparatus may be directly passed to 
the image retrieving apparatus, and retrieval may be executed on-line. 

Figs. 8 to 1 1 show a second embodiment of an image retrieval information 
storing apparatus and an image retrieving apparatus of the present invention. 

Fig. 8 is a configuration diagram representing the second embodiment of the 
image retrieval information storing apparatus of the present invention. The first 
embodiment above is to store the fi'ame feature value of each fi-ame, whereas in the 
present second embodiment, coding information which is the base for extracting the 
featured frame read by coding information reading unit 101 is stored in the prescribed 
storing location of a coding information storing unit 801. By this configuration, what 
is necessary is simply to read the coding information at the prescribed location, when the 
featured fi^ame is to be extracted in image retrieval. This improves efficiency as 
compared with successive reading of the original coded image data. The location for 
storing coding information may be a position corresponding to the coded image data for 
each firame^ for example, as header information of the coded image data for each fi-ame. 
Alternatively, the coding information may be stored independent of coded image data, as 
diflferent data for image retrieval. 

Fig. 9 is a configuration diagram representing the second embodiment of the 
image retrieving apparatus of the present invention. In the image retrieving apparatus 
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in the first embodiment above, the stored frame feature value is input to index 
information generating unit 401. In contrast, in the present second embodiment, frame 
feature value generating unit 102 generating a frame feature value for each frame from 
the input coding information is added and the frame feature value generated by frame 
feature value generating unit 102 is input to index information generating unit 401 . As 
the operation of frame feature value generating unit 102, index information generating 
unit 401 and image retrieval information executing unit 402 is the same as in the first 
embodiment, the operation is not mentioned. 

In the present second embodiment, an image retrieval information storing 
apparatus is implemented in a simple configuration in which only the coding information 
necessary for extracting a featured frame is extracted and stored from the original coded 
image data. In addition, as storage is in a form of the coding information, it is possible 
to change the method of determining a featured frame based on the frame feature value, 
and in addition, to change the very method of calculating the frame feature value for 
extracting the feature frame in the process of retrieval. Thus, more flexible image 
retrieval is possible. 

As in the first embodiment, it is also possible to separately store index 
information of a (predetermined) featured frame together with coding information 
necessary for extracting featured frame by adding, to the image retrieval information 
storing apparatus shown in Fig. 8, frame feature value generating unit 102 generating a 
frame feature value for each frame, index information generating unit 401 generating 
index information of featured frame, and index information storing unit 601 storing the 
index information. 

Fig. 10 shows a configuration diagram of this image retrieval information storing 
apparatus. When coding information and featured frame index information are both 
stored^ a request for switching featured frame in the process of retrieval can be met by 
obtaining the frame feature value based on the coding information, and in a retrieval 
without a request for switching, retrieval at high speed utilizing fixed index information 
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can be performed. 

Fig. 1 1 shows a configuration of an image retrieving apparatus corresponding to 
the image retrieval information storing apparatus in Fig. 10. In the image retrieving 
apparatus in Fig. 1 1, the stored index information and the index information output fi-om 
index information generating unit 401 are both input to image retrieval executing unit 
402 and utilized for image retrieval. 

In addition, in the first embodiment and the second embodiment above, the fi-ame 
feature value for each fi-ame and coding information necessary for extracting the 
featured fi-ame are each stored. These are not incompatible, and therefore the fi-ame 
feature value and the coding information necessary for extracting the featured fi-ame may 
be stored together. Further, the index information, the fi-ame feature value and the 
coding information necessary for extracting the featured fi-ame may be all stored and 
utilized at the time of image retrieval. 

Figs. 12 and 13 show a third embodiment of an image retrieval information 
storing apparatus of the present invention. 

Fig. 12 is a configuration diagram representing the third embodiment of the 
image retrieval information storing apparatus of the present invention. The present 
third embodiment is characterized by including a storing information selecting unit 1201 
selecting which information should be stored as image retrieval information, fi-om among 
index information, the fi-ame feature value and coding information necessary for 
extracting a featured fi'ame. It is noted that the image retrieval information selected by 
storing information selecting unit 1201 is not limited to any one of the above, and 
selection of two or all of the index information, the frame feature value and the coding 
information necessary for extracting a featured fi-ame may also be permitted, depending 
on a situation. 

In the first embodiment, it has been described that the featured frame for a 
motion picture such as a scene change frame and a featured frame related to motion may 
be extracted differently dependent on the manner of determination, and that physical 
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meaning of the frame may differ. For example, a frame in which rapid movement is 
observed and a frame in which movement is very small are frames having completely 
different features. When motion picture retrieval is executed, whether it is appropriate 
to display a frame with rapid movement, or a frame with less movement, or different 
frame having different feature related to motion as the featured frame for the motion 
picture depends on the motion picture which is the object of retrieval, and there may be 
a request for adaptive switching in the process of retrieval. In such a situation, the 
request for switching in the process of retrieval may be immediately met when the image 
retrieval information is stored in the form of frame feature value or in the form of coding 
information, by storing information selecting unit 120L 

Alternatively, when there is some knowledge of the motion picture which is the 
object of retrieval in advance and appropriate featured frame has been known, it is 
possible by storing the image retrieval information in the form of index information of 
the featured frame by storing information selecting unit 1201, to reduce the amount of 
information for image retrieval, and to execute retrieval at high speed. Further, when 
the frame feature value or the coding information, and the index information of the 
featured frame are to be stored together by storing information selecting unit 1201, it is 
possible to meet the request for switching in the process of retrieval immediately, and to 
execute retrieval at high speed. 

Fig. 13 shows the third embodiment of the image retrieving apparatus of the 
present invention. The index information, the frame feature value or the coding 
information necessary for extracting a featured frame stored in the image retrieval 
information apparatus is input to the image retrieving apparatus as appropriate. 

When there is index information, the position of the featured frame is obtained 
based on the index information, the input coded image data is decoded and displayed, 
and the image retrieval is executed. When a new featured frame is required in the 
process of image retrieval and there is the frame feature value, the new featured frame is 
extracted based on the frame feature value, and the index information is generated and 
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used for image retrieval. 

When it is determined that the frame feature value is inappropriate and there is 
coding information, a new frame feature value is generated based on the coding 
information, the new featured frame is extracted from thus generated new frame feature 
value, and index information is generated and used for image retrieval. 

The advantage of storing both the frame feature value and the coding 
information is that image retrieval with such a high degree of freedom is possible. 

When there is coding information and not the input frame feature value, the 
request for extracting a new featured frame is directly issued to frame feature value 
generating unit 102. 

In this manner, in the third embodiment of the image retrieval information 
storing apparatus of the present invention, the information to be stored is selected by 
storing information selecting unit 1201 dependent on the nature of the image which is 
the object of retrieval so that appropriate image retrieval information can be stored and 
adaptive image retrieval can be executed by the image retrieving apparatus. Further it 
is also possible to select the stored information of storing information selecting unit 
1201 dependent on the rate of execution of retrieval or by the restriction of storing 
capacity. 

As to the capacity of image retrieval information, the index information, the 
frame feature value and coding information have capacities larger in this order. For 
example, when the storage capacity for storing image retrieval information is small, only 
the index information having small capacity is stored. In a terminal where the storing 
capacity is sufficiently large and the rate of executing retrieval is slow, all the 
information is stored to ensure degree of freedom of retrieval, and index information is 
frilly made use of so as not to lower the execution rate. 

Further, it is possible to limit the image retrieval information to be stored to only 
two of the index information, frame feature value and the coding information necessary 
for extracting featured frame, and to select the stored information by storing information 
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selecting unit 1201. 

[Effects of the Invention] 

With the configuration as described above, the image retrieval information 
storing apparatus according to the invention described in claim 1 of the subject 
application can store the coding information necessary for extracting the featured frame 
or the firame feature value for each frame, so that it can immediately meet the request for 
switching the featured frame in the process of retrieval. 

With the configuration as described above, the image retrieval information 
storing apparatus according to the invention described in claim 2 of the subject 
application stores the coding information necessary for extracting the featured frame, so 
that the configuration of a storage side can be simplified and a used capacity of a storage 
medium can be reduced. 

With the configuration as described above, the image retrieval information 
storing apparatus according to the invention described in claim 3 of the subject 
apphcation can store the frame feature value necessary for extracting the feature frame, 
so that it can perform retrieval processing at high speed in response to various retrieval 
requests. 

With the configuration as described above, the image retrieving apparatus 
according to the invention described in claim 4 of the subject application can adaptively 
perform image retrieval and perform retrieval processing at high speed based on the 
frame feature value, by extracting the input coding information or the frame feature 
value and a new featured frame in response to a featured frame extraction request issued 
in the process of execution of image retrieval. 

With the configuration as described above, the image retrieving apparatus 
according to the invention described in claim 5 of the subject apphcation can perform 
flexible image retrieval by changing the very method of calculating the frame feature 
value for extracting the featured frame in the process of retrieval when there is a new 
retrieval request for generating the frame feature value based on the input coding 
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information. 

With the configuration as described above, the image retrieving apparatus 
according to the invention described in claim 6 of the subject application can perform 
retrieval processing at high speed for extracting the featured frame based on the input 
frame feature value and achieve a reduced used capacity of a storage medium. 
[Brief Description of the Drawings] 

Fig. 1 represents a first embodiment of an image retrieval information storing 
apparatus of the present invention. 

Fig. 2 represents an internal configuration of a frame feature value generating 
unit in the first embodiment of the image retrieval information storing apparatus of the 
present invention. 

Fig, 3 represents reference relation of frames in prediction coding method. 
Fig. 4 represents a first embodiment of an image retrieving apparatus of the 
present invention. 

Fig. 5 represents an internal configuration of an index information generating 
unit in the first embodiment of the image retrieving apparatus of the present invention. 

Fig. 6 represents the first embodiment of an image retrieval information storing 
apparatus of the present invention. 

Fig. 7 represents the first embodiment of the image retrieving apparatus of the 
present invention. 

Fig. 8 represents a second embodiment of the image retrieval information storing 
apparatus of the present invention. 

Fig. 9 represents a second embodiment of the image retrieving apparatus of the 
present invention. 

Fig, 10 represents the second embodiment of the image retrieval information 
storing apparatus of the present invention. 

Fig. 1 1 represents the second embodiment of the image retrieving apparatus of 
the present invention. 
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Fig. 12 represents a third embodiment of the image retrieval information storing 
apparatus of the present invention. 

Fig. 13 represents a third embodiment of the image retrieving apparatus of the 
present invention. 

Fig. 14 represents an image retrieval information storing apparatus of the 
conventional art. 

Fig. 15 represents an image retrieving apparatus of the conventional art. 

[Description of the Reference Characters] 
1 0 1 J 1401 coding information reading unit 
102, 1402 frame feature value generating unit 
1 03 frame feature value storing unit 

201 prediction mode counting unit 

202 frequency information converting unit 

203 valid motion vector detecting unit 

204 motion vector statistic calculating unit 

205 statistic information converting unit 

401, 1403 index information generating unit 

402, 1501 image retrieval executing unit 

501 scene change frame determining unit 

502 scene change fremie index generating unit 

503 motion featured frame determining unit 

504 motion featured frame index generating unit 
610, 1404 index information storing unit 

801 coding information storing unit 
1 20 1 storing information selecting unit 
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[Document Name] Abstract 
[Abstract] 

[Subject] To provide an image retrieval information storing apparatus and an image 
retrieving apparatus that can quickly and adaptively respond when a different featured 
frame depending on a situation is requested in a process of image retrieval. 
[Solving Means] An image retrieval information storing apparatus is provided with a 
frame feature value generating unit 102 generating a feature value for each frame which 
serves as a direct reference for determining a featured frame based on coding 
information included in coded image data, and a storing unit 103 storing the frame 
feature value as image retrieval information. An image retrieving apparatus is provided 
with an index information generating unit determining a featured frame based on the 
stored feature value for each frame and generating index information based on positional 
information of the featured frame, 
[Selected Drawing] Fig. 1 
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